From NVIDIA Megatron-LM for visibility by RaymondLi0 · Pull Request #18 · bigcode-project/Megatron-LM

RaymondLi0 · 2023-01-24T20:01:13Z

No description provided.

Signed-off-by: oliver könig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

Signed-off-by: oliver könig <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

) Co-authored-by: liaoyang <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Deepak Narayanan <[email protected]>

Signed-off-by: oliver könig <[email protected]>

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: Yuzhong Wang <[email protected]>

Co-authored-by: Claude Sonnet 4.6 <[email protected]>

Signed-off-by: Charlie Truong <[email protected]> Co-authored-by: Kunlun Li <[email protected]> Co-authored-by: Kunlun Li <[email protected]>

Co-authored-by: William Dykas <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: root <[email protected]>

Co-authored-by: Claude Opus 4.6 <[email protected]>

Co-authored-by: Charlie Truong <[email protected]>

Signed-off-by: Keshav Santhanam <[email protected]>

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

Signed-off-by: dimapihtar <[email protected]>

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Charlie Truong <[email protected]>

…form PP (#4775) Co-authored-by: Abhishree Thittenamane <[email protected]> Co-authored-by: root <[email protected]>

Signed-off-by: Keval Morabia <[email protected]>

#4806) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Signed-off-by: Charlie Truong <[email protected]>

…le performance–memory trade-offs (#4181)

…ated MIMO (#4801) Signed-off-by: Kamran Jafari <[email protected]>

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Signed-off-by: Charlie Truong <[email protected]>

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Co-authored-by: Gao Deng <[email protected]>

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Zhongbo Zhu <[email protected]>

Signed-off-by: Charlie Truong <[email protected]>

#4852) Signed-off-by: Cory Ye <[email protected]>

Signed-off-by: jinliangl <[email protected]>

Co-authored-by: Teodor-Dumitru Ene <[email protected]>

…ed offloading (#4692)

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12

RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12

ko3n1g force-pushed the main branch from ad5a627 to cd71d4c Compare February 18, 2026 09:18

ko3n1g and others added 27 commits April 16, 2026 14:00

feat: long convergence resiliency for release tests (#4335)

f098fe8

Signed-off-by: oliver könig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

ci(action): improve GitHub Actions output UX (#4337)

8681ebb

Signed-off-by: oliver könig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

build: bump TransformerEngine to release_v2.14 (#4331)

ceac269

Signed-off-by: oliver könig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

feat: add create-issue skill (#4338)

efbe7a1

Signed-off-by: oliver könig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

Set megatron-fsdp to 0.5.0

97f9ab6

M4 leftover for TE cuda graph (#3137)

01eb7e8

fix: wait for async P2P send before deallocating output tensor (#4047)

260cba7

ci(gb200): add 1-node mr-github functional test variants (#4334)

2aeaf56

Signed-off-by: oliver könig <[email protected]> Signed-off-by: Oliver Koenig <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>

Fix potential coredump issue that occurs when saving a checkpoint (#1871

ded22f4

) Co-authored-by: liaoyang <[email protected]> Co-authored-by: Eric Harper <[email protected]> Co-authored-by: Deepak Narayanan <[email protected]>

docs: bump versions1.json to 0.17.0 (latest) (#4360)

30bc230

Signed-off-by: oliver könig <[email protected]>

Port DeepSeek Sparse Attention to MambaModel (#3553)

a00e944

Add tables and histogram for RL staleness (#4097)

23663a8

[docs] ci: use parent-relative json_url for version picker (#4367)

4ece77d

Signed-off-by: oliver könig <[email protected]>

Fix bug with non-partial rollouts (#3964)

ed5de26

Add QK layernorm support for dot-product attention in MambaModel (#4067)

e15ec3c

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: Yuzhong Wang <[email protected]>

Docs: improve docstrings and comments in example training loop (#4041)

75a2878

Update copy-pr-bot.yaml [skip ci]

86b7218

feat(ckpt): add --async-ckpt-use-cpu-shm argument (#4355)

9978968

Co-authored-by: Claude Sonnet 4.6 <[email protected]>

cp: Fix UT timeout (#4310) (#4373)

664baa8

Signed-off-by: Charlie Truong <[email protected]> Co-authored-by: Kunlun Li <[email protected]> Co-authored-by: Kunlun Li <[email protected]>

Fix RL reward due to stop token (#4096)

e4d3a4c

FA4 Inference (#4186)

76ac7c2

Co-authored-by: William Dykas <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: root <[email protected]>

Make param_index_map always use unpacked (full numel) offsets (#4328)

3315c86

Co-authored-by: Claude Opus 4.6 <[email protected]>

Add activation logging and tokens per expert logging (#3842)

8be1e79

Co-authored-by: Charlie Truong <[email protected]>

Fix RL to once again work with --skip-train (#4249)

98a51eb

Fix Megatron initialization with extra_args_provider (#4327)

afae25b

Signed-off-by: Keshav Santhanam <[email protected]>

Rename MambaModel/MambaStack to HybridModel/HybridStack (#4099)

15e07a2

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

chore(beep boop 🤖): Bump (main) (2026-04-20)

3046182

dimapihtar and others added 30 commits May 18, 2026 16:18

Fix no nvrx tests (#4847)

5096459

Signed-off-by: dimapihtar <[email protected]>

Thread custom process groups through MoE grad finalization (#4782)

aa786b7

Fix unit tests (#4689)

648b916

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Tests/dynamic inference functional coverage (#4761)

6c3fa6d

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Charlie Truong <[email protected]>

Update copy-pr-bot.yaml [skip ci]

74daec9

Fix oncall references (#4722)

952efb4

Update golden values for nightly functional tests (#4850)

b2a8ec7

fix(inference): size DynamicInferenceContext KV layer_map for non-uni…

43fb2f5

…form PP (#4775) Co-authored-by: Abhishree Thittenamane <[email protected]> Co-authored-by: root <[email protected]>

Modernize post-training modelopt example scripts (#4807)

14aaa7e

Signed-off-by: Keval Morabia <[email protected]>

chore: rotate oncall schedule

38986a9

test: add inference performance test harness for GPT 583M, hybrid 2B,… (

23ba357

#4806) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

ci: Prevent shell trace in parts of _run_training.sh (#4884)

1ffcd1d

Signed-off-by: Charlie Truong <[email protected]>

Ignore Vim swap files (#4860)

14dabeb

M-FSDP: Make fine_grained_param_gather configurable for MXFP8 to enab…

b5d143f

…le performance–memory trade-offs (#4181)

MimoOptimizer: fix distributed checkpoint save and load for non-coloc…

68d64f7

…ated MIMO (#4801) Signed-off-by: Kamran Jafari <[email protected]>

Update copy-pr-bot.yaml [skip ci]

5df3e4e

Route non-Muon params through DistributedOptimizer (#4771)

0044db1

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

ci: Gate optional CI jobs with repository variables (#4907)

990fa69

Signed-off-by: Charlie Truong <[email protected]>

Fix tokenizers bug in nightly (#4833)

e69a1ea

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Allow optimizer CG to share the same pool as full-iter CG (#4698)

daec17c

Use sharded_state_dict_default in MLP.sharded_state_dict (#4693)

32a7e46

Co-authored-by: Gao Deng <[email protected]>

Fix MTP recompute crash with packed sequences (#4593)

2b77d32

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Zhongbo Zhu <[email protected]>

Update PR template (#4904)

c6096d9

ci: Update perf test to output logs for tests to pass (#4906)

0588bf7

Signed-off-by: Charlie Truong <[email protected]>

Also persist asymmetrical units for the MXFP8 transpose weight buffer. (

6576040

#4852) Signed-off-by: Cory Ye <[email protected]>

fix no_shard training convergency and add unittest for no_shard (#3754)

80a2d39

Signed-off-by: jinliangl <[email protected]>

Move policy epoch stats to the message object (#4533)

e9a3184

Co-authored-by: Teodor-Dumitru Ene <[email protected]>

Add a knob to throttle the max allowed inflight offload in fine grain…

9a7cd17

…ed offloading (#4692)

refactor(data): consolidate get_batch and enable PP for SFT THD (#4103)

5e4fc93

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Allow YAML MoE configs to use model specs (#4822)

0d198cd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

From NVIDIA Megatron-LM for visibility#18

From NVIDIA Megatron-LM for visibility#18
RaymondLi0 wants to merge 7000 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants